XG Boost in Mental Health Classification

Fall 2025 Capstone for Data Science

Aabiya Mansoor, Abigail Penza Jackson, Corina Rich, Madelyn Champion (Advisor: Dr. Cohen)

2025-11-30

Introduction

eXtreme Gradient Boosting (XGB) algorithm

  • A supervised machine learning algorithm
  • Modification of Gradient Boosting Framework
  • Ensemble of weak decision trees
  • L1,L2 regularization
  • High performance, speed, scalability
  • Applications in healthcare, education, public health, finance, and engineering.

Introduction

XGBoost for mental disorder classification

  • Using XGBoost on clinical and survey data
  • Classification of individuals into one of four categories: Bipolar I Disorder, Bipolar II Disorder, Major Depressive Disorder, and Normal
  • XGBoosts proves to be a suitable algorithm for the problem

Literature Review

eXtreme Gradient Boosting (XGBoost) in healthcare

  • XGBoost strengths in real world problems handling class imbalance, heterogeneous data types, or non-linear relationships. (Chen and Guestrin 2016)
  • XGBoost combined with DL for breast cancer classification with high reliability. (Liew, Hameed, and Clos 2021)
  • XGBoost with biomarker data to improve depression diagnoses in a large Dutch population dataset.(Sharma and Verbeke 2020)
  • XGBoost to multi-modal datasets to predict self-harm in young adults.(Xu et al. 2024)
  • A hybrid algorithm (XGBoost-HOA) to classify depression, anxiety, and stress
  • Dual XGBoost models applied to distinguish between deficit and non-deficit schizophrenia subtypes using fMRI features.(Zhang, Jia, and Shang 2022)
  • XGBoost compared with linear regression in predicting depression among refugee children.(Saleh et al. 2024)

Literature Review

XGBoost limitations for imbalance data

Literature Review

XGBoost in Education

XGBoost in Public health

  • XGBoost used to predict daily COVID-19 cases in the United States.(Fang et al. 2022)
  • A hybrid model integrating XGBoost, Random Forest, and Antlion Optimization used to predict infectious disease outbreaks.(Sivakumar and Elangovan 2023)

Literature Review

XGBoost in financial sector

  • XGBoost model’s ability to rank feature importance and prevent overfitting using regularization made it a top performer in credit risk prediction tasks(Li et al. 2020)
  • XGBoost to forecast volatility in the U.S. stock market, identifying the Economic Policy Uncertainty Index as a critical predictor.(Fomunyam 2023)

XGBoost In pharmaceutical research

Literature Review

XGBoost in educational diagnostics

XGBoost in Sports Analytics

Methods: Extreme Gradient Boosting (XGBoost)

  • Ensemble of weak decision trees
  • Trained sequentially, each tree corrects previous errors
  • Captures non-linear relationships within the 17-variable dataset
  • Uses gradient descent to minimize loss
  • Includes regularization to prevent overfitting

Methods: XGBoost Formulas

  • Loss term: Measures prediction error
  • Regularization term: Penalizes model complexity
  • Ensures balance between accuracy and generalization

\[ \text{Obj} = \sum_{i=1}^{n} L(y_i, \hat{y}_i) + \sum_{k=1}^{K} \Omega(f_k) \]

Methods: Training the Workflow

  • Stage 1: Initial Prediction
  • Stage 2: Iterative Boosting (Tree k)
  • Stage 3: Final Ensemble Prediction

XG Boost

Data Exploration and Visualization

  • Sourced from Kaggle (2023 clinical records)
  • 120 individuals assessed for mental disorders -Labels:
    • Normal (25% of total)
    • Bipolar Type I (23% of total)
    • Bipolar Type II (26% of total)
    • Depression (26% of total)
  • 17 diagnostic variables
  • Mix of clinical, behavioral, demographic factors

Data Exploration and Visualization

  • Examined class distribution
  • Identified potential imbalance or outliers
  • Explored correlations between features

Cramers V

Modeling and Results

Data Preprocessing

To facilitate analysis, all categorical variables in the dataset were transformed into numerical or grouped categorical formats in Excel. The transformations were applied as follows:

  • Ordinal responses (Sadness, Euphoric, Exhausted) recoded:
    • Seldom=1, Sometimes=2, Usually=3, Most-Often=4
  • Binary items (Suicidal Thoughts, Mood Swing):
    • Yes=1, No=0
  • 1–10 rating scales (Sexual Activity, Optimism) grouped:
    • 1–3 → Cat 1, 4–6 → Cat 2, 7–9 → Cat 3

Modeling and Results

Hyperparameter Optimization

  • The dataset was randomly partitioned into an 80/20 training–testing split.
    • This split resulted in 96 patient surveys used for model training and 24 surveys reserved for independent testing.
    • The predictive target for all models was the expert clinical diagnosis
  • An XGBoost model specification was created using the tidymodels framework where all major hyperparameters were set to be tuned including:
    • the number of trees
    • tree depth
    • learning rate
    • loss reduction,
    • minimum node size
    • subsampling rate
    • number of predictors considered at each split

Modeling and Results

Hyperparameter Optimization

  • Five-fold stratified cross-validation was used for hyperparameter tuning.
    • Where the training dataset was partitioned into five equally sized folds
    • For each tuning iteration:
      • Models were trained on four folds
      • Validated on the remaining fold
      • Which ensured stable and unbiased performance estimates

Five Fold Validation(Singh 2024)

Modeling and Results

Hyperparameter Optimization

  • A 30-point hyperparameter grid was generated using a space-filling design.
  • Model performance was evaluated using Accuracy and ROC AUC.
  • Performance improved with more boosting iterations and depth = 2, peaking at ~625 trees with Accuracy > 0.80 and ROC AUC > 0.90.

VisulizationTuningResults

Modeling and Results

Hyperparameter Optimization

  • The select_best() function from the tidymodels framework was used to systematically extract the hyperparameter combination that achieved the highest ROC AUC across all resampled evaluations.

  • The resulting optimal configuration included 3 predictors (mtry = 3), 627 boosting iterations, a minimum node size (min_n) of 2, a learning rate of 0.16501, a loss-reduction parameter of 0.0034, and a subsample proportion of 0.5655.

    • This combination represents the tuning grid entry labeled pre0_mod06_post0, which achieved the best balance between model complexity and predictive performance during cross-validation.

BestParams

Modeling and Results

Model Evaluation & Performance

Prediction

The model’s predictions on the held-out test set showed strong agreement with the expert diagnoses. Specifically:

  • Bipolar Type-1: 4 of 6 cases were correctly predicted (66.7%)
  • Bipolar Type-2: 5 of 7 cases were correctly predicted (71.4%)
  • Depression: 6 of 7 cases were correctly predicted (85.7%)
  • Normal: 4 of 6 cases were correctly predicted (66.7%)

Modeling and Results

Model Evaluation & Performance

  • XGBoost shows moderate overall predictive performance with strong discriminative ability.
  • Accuracy: 0.654 — correctly classifies ~65% of observations.
  • Kappa: 0.534 — indicates moderate agreement beyond chance.
  • Macro ROC AUC: 0.927 — strong ability to rank observations across all classes.
  • Overall: High AUC but moderate accuracy/kappa, suggesting strong discrimination but room to improve final classification.

Modeling and Results

Feature Importance & Interepretability

  • SHAP analysis shows both broadly influential and class-specific predictors across the four classes.

  • Most Influential (Across Classes):

    • Mood Swing: strongest overall, especially for Bipolar I & II
    • Suicidal Thoughts: key for Depression and Bipolar II
    • Sadness: major driver for Bipolar I
  • Class-Specific Influences:

    • Concentration (Cat 2): strong indicator for Normal
    • Authority & Respect: most impactful for Bipolar II
    • Ignore & Move On and Exhausted: help distinguish the Normal class
  • Conclusion: Distinct SHAP patterns reveal unique psychological signatures across classes, reflecting a well-calibrated multiclass XGBoost model.

Modeling and Results

Sensitvity & Robustness Analysis

  • Sensitivity analysis shows the model is stable and consistent across resampling.
    • Mean Accuracy: 0.804 — strong predictive performance.
    • SD Accuracy: 0.0821 — moderate variability across samples.
    • Mean ROC AUC: 0.957 — excellent and consistent discriminative ability.
    • SD ROC AUC: 0.0397 — highly stable class separation.
  • Overall: High, stable AUC and low–moderate variability, indicating robust performance with slight accuracy sensitivity to data shifts.

Modeling and Results

Summary of Modeling Results

  • On the 24-case test set, the model showed strong agreement with expert diagnoses and good generalization.

  • It correctly identified most cases in each diagnostic category.

  • Final model strengths:

    • High discriminative power: ROC AUC > 0.90 across CV and sensitivity analyses
    • Stable generalization: low variability across repeats
    • Interpretability: clear clinical insights via SHAP and feature importance
    • Accurate predictions: strong performance across all four diagnostic groups
  • Conclusion: XGBoost is an effective, interpretable model for mental health classification in a small structured dataset, offering strong predictive value and clinically meaningful insights.

Conclusion

Study Purpose

  • Evaluate effectiveness of XGBoost for predicting mental health outcomes.
  • Develop, tune, and validate the model.
  • Identify highest‑predictive features and assess ability to capture non‑linear relationships.
  • Model showed strong accuracy and generalizability.

Conclusion

Key Findings

  • Top predictors: stress indicators, behavioral patterns, wellness factors.
  • High feature importance suggests relevance to mental‑health risk.
  • Model effectively distinguished between classes.
  • Stable performance across tuning configurations with minimal overfitting.

Implications

  • XGBoost shows potential for early warning mental‑health monitoring.
  • Useful for identifying at‑risk individuals based on survey data.
  • Self‑reported data may introduce bias—interpret with care.

Conclusion

Future Work

  • Use larger and more diverse datasets.
  • Incorporate behavioral or time‑based variables.
  • Compare with Random Forest, LightGBM, neural networks.
  • Test in real‑world settings.
  • Address fairness and ethical use of health data.

Conclusion: Overview

  • Overall, XGBoost proved to be an effective and interpretable modeling approach for mental health classification in a small structured dataset, delivering both predictive value and meaningful interpretability essential for clinical research. Our XGBoost Model performed with 88% accuracy. Machine-learning approaches may provide timely insights that complement traditional assessment methods and support early identification of individuals who may benefit from targeted intervention. With further refinement and validation, predictive modeling has the potential to become an increasingly valuable component of mental-health research and practice.

References

Chen, Tianqi, and Carlos Guestrin. 2016. “XGBoost: A Scalable Tree Boosting System.” In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–94. New York, NY, USA: ACM; Association for Computing Machinery. https://doi.org/10.1145/2939672.2939785.
Fang, Zhiyuan, Shuang Yang, Chun Lv, et al. 2022. “Application of a Data-Driven XGBoost Model for the Prediction of COVID-19 in the USA: A Time-Series Study.” BMJ Open 12 (7): e056685. https://doi.org/10.1136/bmjopen-2021-056685.
Fomunyam, R. A. 2023. “The Impact of the u.s. Macroeconomic Variables on the CBOE VIX Index.” Journal of Economics and Finance 47 (1): 77–94. https://www.proquest.com/docview/2642416749.
Hakkal, Soukaina, and Ayoub Ait Lahcen. 2024. “XGBoost to Enhance Learner Performance Prediction.” Computers and Education: Artificial Intelligence 7: 100254. https://doi.org/10.1016/j.caeai.2024.100254.
Hu, Ting, and Ting Song. 2019. “Research on XGBoost Academic Forecasting and Analysis Modelling.” Journal of Physics: Conference Series 1324 (1): 012091. https://doi.org/10.1088/1742-6596/1324/1/012091.
Li, H., Y. Cao, S. Li, J. Zhao, and Y. Sun. 2020. XGBoost Model and Its Application to Personal Credit Evaluation.” IEEE Intelligent Systems 35 (3): 52–61. https://doi.org/10.1109/MIS.2020.2972533.
Liew, Xin Yu, Nazia Hameed, and Jeremie Clos. 2021. “An Investigation of XGBoost-Based Algorithm for Breast Cancer Classification.” Machine Learning with Applications 6: 100154. https://doi.org/10.1016/j.mlwa.2021.100154.
Nikolaidis, P. T., Beat Knechtle, and other co-authors. 2023. “Analysis of the 10-Day Ultra-Marathon Using a Predictive XGBoost Model.” Open Sports Sciences Journal 16. https://uwf-flvc.primo.exlibrisgroup.com/discovery/fulldisplay?docid=cdi_doaj_primary_oai_doaj_org_article_986cc6e5973948ed919ab7ac5176113a.
Saleh, M., E. Amona, M. Kuttikat, I. Sahoo, D. Chan, J. Murphy, and M. Lund. 2024. “Child Mental Health Predictors Among Camp Tamil Refugees: Utilizing Linear and XGBOOST Models.” PLoS ONE 19 (9): e0303632. https://doi.org/10.1371/journal.pone.0303632.
Sharma, Anjali, and Wouter J. M. I. Verbeke. 2020. “Improving Diagnosis of Depression with XGBOOST Machine Learning Model and a Large Biomarkers Dutch Dataset (n = 11,081).” Frontiers in Big Data 3: 15. https://doi.org/10.3389/fdata.2020.00015.
Singh, Aarushi. 2024. “Cross-Validation: Ensuring Model Robustness — Image.” Online; image from the article “Cross-Validation: Ensuring Model Robustness.” https://engineersplanet.com/cross-validation-ensuring-model-robustness/.
Sivakumar, R., and S. Elangovan. 2023. “Prediction of Seasonal Infectious Diseases Based on Hybrid Machine Learning Approach.” International Journal of Health Sciences 7 (2): 1958–69. https://research.ebsco.com/c/imx7og/viewer/pdf/ff4a3en7vb.
Su, Wenjie, Fei Jiang, Chen Shi, Dapeng Wu, Lihua Liu, Shu Li, Ying Yuan, and Jie Shi. 2023. “An XGBoost-Based Knowledge Tracing Model.” International Journal of Computational Intelligence Systems. https://doi.org/10.1007/s44196-023-00192-y.
Wiens, Mark, April Verone-Boyle, Nate Henscheid, J. T. Podichetty, and John Burton. 2025. “A Tutorial and Use Case Example of the eXtreme Gradient Boosting (XGBoost) Artificial Intelligence Algorithm for Drug Development Applications.” Clinical and Translational Science 18: e70172. https://doi.org/10.1111/cts.70172.
Xu, Xiao-Ming, Yang S. Liu, Su Hong, Chuan Liu, Jun Cao, Xiao-Rong Chen, Zhen Lv, et al. 2024. “The Prediction of Self-Harm Behaviors in Young Adults with Multi-Modal Data: An XGBoost Approach.” Journal of Affective Disorders Reports 16: 100723. https://doi.org/10.1016/j.jadr.2024.100723.
Zhang, Ping, Yibing Jia, and Yanan Shang. 2022. “Research and Application of XGBoost in Imbalanced Data.” International Journal of Distributed Sensor Networks 18 (6). https://doi.org/10.1177/15501329221106935.